When Natural Language is Better than Menus: A Field Study

Whereas many natural language researchers have identiied ways in which natural language systems need to be augmented in order to make current natural language interactions more natural, most evaluations of natural language interfaces have failed to study natural language in the context of its use. Since there are now restricted natural language systems available commercially, we feel that it is critical to identify whether in fact these systems are of beneet to end-users and what beneets these limited natural language systems ooer. In this study, we have evaluated a restricted NL as an interface for database access , and compared it with another type of database interface medium, provided by a menu system. We feel that this is an important component of this study: evaluation cannot done in a vacuum, since for most information access applications there will be other types of interfaces available. We identiied a group of users who made persistent use of the Natural Language system. It seemed that they were exploiting the fact that it allowed them to retrieve information at various levels of granularity, often by the use of coordinating expressions. They also seemed to take advantage of the increased exibility of expression provided by the NL interface, exempliied by the use of ellipsis, and the capacity to vary the presentation of output. We also analysed the learnability of a restricted subset of Natural Language by examining the errors of both persistent and non-persistent users of the system. We discuss the implications of this analysis for determining those features of applications that make NL a suitable interface. We also comment on how users of NL systems might best be trained.